In recent years, there has been an increasing demand for measuring a distance from an intended position to an object or a three-dimensional position (shape) of the object. For example, there has been proposed for a visibility support for a vehicle driver, a method for avoiding collision with surroundings more precisely by not only presenting an image of the surroundings captured by a compact camera mounted on a vehicle, but also concurrently presenting accurate information on the distance between the vehicle and the object measured by a distance measuring apparatus. Further, input and output devices for a three-dimensional image are in demand for mobile phones and television sets to reproduce more realistic images. The distance measuring apparatuses for the above-described applications are required to be miniaturized and have high precision.
A conventionally used method for measuring a distance to an object or a three-dimensional position of an object includes a stereo distance-measuring method that utilizes a principle of triangulation. With the stereo distance-measuring method, a distance to an object is calculated based on a disparity that occurs between plural cameras.
FIG. 30 is a diagram that describes an example of distance calculation to an object using the stereo distance-measuring method in the case where two cameras, a camera a and a camera b are used. Rays 101a and 101b of an object 100 form images on an imaging regions 104a and 104b via optical centers 105a and 105b of a lens 102a of a camera a and a lens 102b of a camera b, respectively. An optical axis 103a and an optical axis 103b represent an optical axis of each camera. In this case, when the object 100 forms an image at a position 107a that is away by Pa from an intersection 106a of the imaging region 104a and the optical axis 103a in the camera a, and at a position 107b that is away by Pb from an intersection 106b of the imaging region 104b and the optical axis 103b in the camera b, for example, a disparity P (=Pb−Pa) occurs between the camera a and the camera b. The disparity P changes depending on a distance D between the distance measuring apparatus and the object. When the optical axis 103a of the camera a and the optical axis 103b of the camera b are parallel to each other, the distance therebetween is a baseline length B, and a focal distance of the camera a and the camera b is f, the distance D to the object is expressed by (Equation 1). Therefore, when the baseline length B and the focal distance f are known by calibration processing and the like in advance, the distance D to the object 100 can be calculated by obtaining the disparity P.
                    [                  Equation          ⁢                                          ⁢          1                ]                                                            D        =                  f          ⁢                                          ⁢                      B            P                                              Equation        ⁢                                  ⁢        1            
It is to be noted that the optical axes of the camera a and the camera b are not parallel to each other in many cases in a real environment. Therefore, rectification as shown in Non-patent reference 1 is performed. This produces a picture in which the optical axes are parallel, and it is known that the distance D can be calculated by using operation using the above (Equation 1).
The imaging regions 104a and 104b are configured generally by an imaging device such as CCD (charge coupled device) and CMOS (complementary metal oxide semiconductor). Therefore, the disparity P is calculated by using a brightness signal of an object image that has been discretized on a two-dimensional plane, so that a disparity detection resolution is one pixel in general. A distance measuring resolution (hereinafter referred to as a measuring accuracy) is determined from the relationship of (Equation 1) using the disparity detection resolution.
Further, the three-dimensional position of an object can be calculated by using a method described below with reference to FIG. 31 to FIG. 33, for example.
FIG. 31 is a diagram which shows a positional relationship of the distance measuring apparatus and the object. In FIG. 31, the original point Mw (0, 0, 0) of the world coordinate is the optical center 105a of the camera a. As shown in the diagram, when a three-dimensional position coordinate of a point 111 in the object 110 is Mw (Xw1, Yw1, Zw1), Zw1 is derived from the distance D calculated using (Equation 1) described with reference to FIG. 30.
FIG. 32 is a diagram which shows the camera a and the point 111 on the object viewed from the direction of a minus side of Yw axis. As shown in the diagram, when the intersection 106a is an original point ms (0, 0) of a two-dimensional coordinate system of the imaging region 104a, Xw1 can be expressed as (Equation 2) by using the coordinate ms (xs1, ys1) of an imaging position 107a.
                    [                  Equation          ⁢                                          ⁢          2                ]                                                                      X          ⁢                                          ⁢          w          ⁢                                          ⁢          1                =                  xs          ⁢                                          ⁢          1          ⁢                                          ⁢                                    Zw              ⁢                                                          ⁢              1                        f                                              Equation        ⁢                                  ⁢        2            
FIG. 33 is a diagram which shows the camera a and the point 110 on the object viewed from the direction of a plus side of Xw axis. In the same manner as in FIG. 32, when the intersection 106a is the original point ms (0, 0) of the two-dimensional coordinate system of the imaging region 104a, Yw1 can be expressed as (Equation 3) by using the coordinate ms (xs1, ys1) of an imaging position 107a
                    [                  Equation          ⁢                                          ⁢          3                ]                                                                      Y          ⁢                                          ⁢          w          ⁢                                          ⁢          1                =                  ys          ⁢                                          ⁢          1          ⁢                                          ⁢                                    Z              ⁢                                                          ⁢              w              ⁢                                                          ⁢              1                                                                                  ⁢              f                                                          Equation        ⁢                                  ⁢        3            
A measuring accuracy of the three-dimensional position is also determined by the disparity detection resolution as well as the measuring accuracy described above.
The following describes a specific example of a method of calculating the disparity P. A SAD (sum of absolute difference) that is a correlation value of each sub region of each of the images is calculated for the picture a captured in the imaging region 104a of the camera a and the picture b captured in the imaging region 104b of the camera b, and then the disparity P; that is, an image shift for each sub region between the picture a and the picture b is calculated by using the calculated correlation value. It is to be noted that the SAD is an example of the correlation value, and generally known SSD (sum of squared difference) and NCC (normalized cross-correlation) and the like can be used as the correlation value as well.
The following describes calculation of disparity that uses a SAD that is a representative correlation value with reference to FIG. 34 to FIG. 36.
FIG. 34 is a diagram that explains a method of expressing brightness of each pixel in a picture. As shown in the diagram, the gradation of brightness in each pixel is expressed by the density of lines, where 0 is black and 15 is white. The brightness may be a value after decimal point.
FIG. 35A is a diagram which illustrates a portion of a picture a when a texture of a imaged object is viewed from the direction of the object. FIG. 35B is a diagram which illustrates a portion of a picture b when a texture of an imaged object is viewed from the direction of the object.
In an image block b surrounded by a bold line in FIG. 35B, an image identical to the image in an image block a surrounded by a bold line in FIG. 35A appears when the object is at infinity. A disparity occurs in the case where the object is within a finite distance as shown in FIG. 35A and FIG. 35B, so that the image of FIG. 35B appears on the right with respect to the image in FIG. 35A. The case where FIG. 35A and FIG. 35B have actual disparity of 3.6 pixels will be described here. The image block b is shifted by one pixel in the right direction from the position indicated by the bold line in FIG. 35B in order to search for an image region which has the highest correlation with the image block a, and the SAD is calculated based on (Equation 4) for each shifting amount.[Equation 4]SAD=Σ|Ia(i,j)−Ib(i,j)|  Equation 4
Here, Ia and Ib represent a brightness value of respective image blocks, and i and j represent local address of respective image blocks. The image block a and the image block b are the same image size, and a sum of absolute values, in the block, of brightness difference at the same address in both image blocks is calculated for each shifting amount. The shape of the image block is explained as square here, however, the shape of the image block may be rectangular or correspond to the characteristic of the texture.
FIG. 36 is a diagram which illustrates a transition of the SAD when the image block b is shifted by one pixel at a time. The SAD is the smallest when the shifting amount is 4 pixels, and thus the correlation between the image block a and the image block b is considered to be the highest. Therefore, the disparity of the camera a and the camera b in the image block a is calculated as 4 pixels, and the disparity P of (Equation 1) is obtained by multiplying the calculated disparity by an pixel pitch size, so that the distance D to the object can be calculated. Although the disparity close to the actual disparity of 3.6 pixels can be obtained in this case, a disparity with the accuracy of pixels after the decimal point (hereinafter referred to as a sub-pixel disparity) cannot be obtained because the accuracy of disparity detection is one pixel.
A method of estimating a disparity of a sub-pixel level is proposed as a method to obtain a disparity with a measuring accuracy, that is, the distance measuring resolution of higher accuracy than a one pixel unit (see, for example, Patent Reference 1). With the method of estimating a sub-pixel disparity called equiangular linear fitting, for example, an actual disparity is estimated at the sub-pixel level through linear interpolation by assuming the SAD transition has the same tilt angle of θ in both the right and left side with respect to the actual disparity as shown in FIG. 37. An equation for calculating a sub-pixel disparity of an equiangular linear fitting, that is, an interpolation function is indicated by (Equation 5).
                    [                  Equation          ⁢                                          ⁢          5                ]                                                                      P          =                                                    P                ⁢                                                                  ⁢                min                            +                                                d                  ·                  in                                ⁢                                                                  ⁢                the                ⁢                                                                  ⁢                case                ⁢                                                                  ⁢                of                ⁢                                                                  ⁢                R                ⁢                                                                  ⁢                                  (                  1                  )                                                      <                          R              ⁡                              (                                  -                  1                                )                                                    ⁢                                  ⁢                  d          =                      0.5            ⁢                                                  ⁢                                                                                R                    ⁢                                          (                      1                      )                                                        -                                      R                    ⁡                                          (                                              -                        1                                            )                                                                                                            R                    ⁡                                          (                      0                      )                                                        -                                      R                    ⁡                                          (                                              -                        1                                            )                                                                                  ·              in                        ⁢                                                  ⁢            other            ⁢                                                  ⁢            cases                          ⁢                                  ⁢                  d          =                      0.5            ⁢                                                            R                  ⁢                                      (                    1                    )                                                  -                                  R                  ⁡                                      (                                          -                      1                                        )                                                                                                R                  ⁡                                      (                    0                    )                                                  -                                  R                  ⁡                                      (                    1                    )                                                                                                          Equation        ⁢                                  ⁢        5            
Here, P is a sub-pixel disparity, Pmin is an shifting amount when the SAD is the smallest (integer disparity), R(0) is a correlation value (SAD) with the shifting amount having the smallest SAD, and R(−1) and R(1) are a SAD with adjacent shifting amount.
In addition, a method of interpolation calculation for an actual disparity through a linear function of higher order such as a quadratic function or a nonlinear function by assuming that a transition of correlation value such as the SAD for each shifting amount with the actual disparity being the basis is proposed. In the case of an object in which the brightness changes linearly as shown in FIG. 35A and FIG. 35B, the SAD transition becomes symmetrical with the actual disparity being the basis as shown in FIG. 36, and the SAD changes linearly. Therefore, the disparity of 3.6 pixels can be obtained accurately by performing sub-pixel disparity estimation by using equiangular linear fitting.
Patent Reference 1: Japanese Unexamined Patent Application Publication No. 2000-283753
Non-Patent Reference 1: “Sanjigen Bijon” Jo Go, Tsuji Saburo, Kyoritsu Shuppan Co., Ltd., pp 96 to 99, published on Sep. 25, 2002